The invention relates to obtaining different views of images in an image-based virtual environment.
Technologies for rendering high-quality graphics and images in real time are important to the success of virtual reality. In virtual reality applications, users can navigate freely within computer synthesized worlds. To render virtual worlds, computers are used to transform the three dimensional coordinates of objects to a two dimensional coordinate space view plane (e.g., computer screen). Inevitably, the power of computers restricts the complexity of virtual worlds being designed.
For this reason, another approach, termed "image-based rendering technology", was proposed to reduce the impact of computing power on the design of virtual worlds. This new approach uses an image or collection of images, called an environment map, which characterizes the appearance of a scene when viewed from a particular position. The environment map contains the pixel values used to display a scene. The image environment map is first wrapped onto an object surface having a certain geometry, such as a cube, a sphere, or a cylinder. Afterwards, by locating the viewing position at the geometrical center of the wrapped object, perspective-corrected views of scenes can be reconstructed from the image on the object surface during playback. For example, in an article written by Greene, entitled "Environment Mapping and Other Applications of World Projections," IEEE Computer Graphics and Applications, Vol. 6, No. 11, pages. 21-29, November 1986, six images on the faces of a cube are used as the environment map.
The basic technique for producing the perspective-corrected scene is to project a desired portion of the wrapped environment map onto a view plane, for example, the computer screen. The mathematical mapping from the wrapped environment map to the view plane depends on the geometry of the object surface and the position of the view plane in world space.
The approach provided by image-based virtual reality technology is well-suited for building virtual worlds having complex backgrounds because the time needed to render the image-based world is generally independent of the complexity of the virtual world.
One type of environment map is the spherical map. Spherical mapping mechanisms are described in U.S. Pat. Nos. 5,359,363, 5,384,588, 5,313,306, and 5,185,667, all of which are incorporated herein by reference. The major advantage of spherical mapping systems is that the environment map is able to provide users with 360 degrees of both horizontal and vertical pannings of views within the mapped sphere. However, a common problem with spherical mapping systems relates to image acquisition. Generally, to acquire the image for a spherical environment map, users usually require special and relatively expensive cameras having fish-eye lenses.
The unlimited range of panning with spherical mapping systems is an attractive feature. However, this feature is not always necessary in certain virtual reality applications. For example, it may not be necessary to have an unlimited vertical field of view. Thus, cylindrical mapping systems are often used to view an environment map. Cylindrical mapping systems provide 360 degrees of horizontal panning, but only a limited range of vertical panning. A method and apparatus for generating perspective views of a scene from a cylindrical environment map are described in U.S. Pat. No. 5,396,583, issued to Chen et al. (hereafter Chen), which is incorporated herein by reference. The device developed by Chen allows the user to pan the viewing window horizontally without limit and vertically within a limited range.
With a cylindrical mapping system, users can generate an image environment map by stitching together the pictures taken at surrounding scenes with a rotatable camera fixed at a tripod. Digital image processing software is available for accomplishing the stitching process.
Rendering an environment map can also be accomplished using a technique called "texture mapping". In texture mapping, a digitized or synthesized image texture is mapped onto a surface represented by surface values (u,v). For example, with a spherical environment map, the sphere is first rendered into a lookup table which is the same size as the final image. In other words, the table includes a (u,v) entry for every pixel. The "u" index is along the equator; while the "v" index is between the poles. Conceptually, this process is analogous to placing a decal on a solid object. The rectangular texture image resides in its own (u,v) coordinate space, and the surface resides in the three-dimensional world space (X.sub.w,Y.sub.w,Z.sub.w). When rendered on a view plane (e.g., computer screen), the world space coordinates of the surface are transformed to screen coordinates (X.sub.s,Y.sub.s).
In virtual reality applications, users usually prefer the ability to freely navigate to obtain different views of scenes. However, the large number of coordinate transformations for doing so require substantial computation power, thereby significantly degrading the performance of real-time playback on low-cost personal computers. For this reason, current panoramic image-based virtual reality applications usually fix the users' viewing position at the center of the panoramic image to simplify the coordinate transformations. Users can pan the view directions vertically and horizontally from this viewing position.
Referring to FIG. 1, a side view of a conventional process, known as dewarping for projecting an object surface 10 onto a view plan 12, is shown. The radius of a cylinder 14 upon which a panoramic image is displayed is denoted as r, and the distance from a viewing position 16 to the view plane 12 is denoted as d. The radius r can be easily calculated by dividing the circumference of the cylinder by 2.pi.. The vertical rotation angle of the view plane 12 is denoted as .phi.. Horizontally rotating the view plane 12 around the panoramic image on the cylinder 14 achieves the horizontal view rotation. Vertically rotating the view plane 12 achieves vertical panning of the view. Changing the distance d between the viewing position 16 and the view plane 12 achieves the zoom-in or zoom-out operations. Because d and .phi. remain unchanged, the geometry of the projected portion of the panoramic image remains the same for any longitude.
The scenes also can be zoomed in or out by changing distance d between the viewing position and the view plane. However, even with this restriction on the viewing position, there are still a large number of computations needed when the user continuously changes browsing conditions. Fortunately, for cylindrical panoramic mapping systems, the coordinate transformations can be further simplified due to the regularity of the cylinder geometry. Moreover, certain accelerating techniques can be applied to their coordinates' transformations. Chen, for example, describes a table-lookup technique which includes recording the projected position, e.g., starting point and ending point, for each vertical (or horizontal) scanline of a view plane. The table-lookup technique is based on scanline coherence of a cylindrical environment map. By scanline coherence, it is meant that the relationship between the scanlines in the cylindrical environment and those in the view plane have a regular or ordered relationship. Thus, mappings between portions of the environment map and view plane may be performed by simple scaling calculations. Basically, the entries in a lookup table include the texture coordinate (u,v) values, for the starting point of each vertical scanline on the view plane. Due to the coherence of the cylindrical environment map, the lookup table need not be recalculated during horizontal view rotations. The only requirement is to offset or scale the u value by a constant value for each scanline to achieve desired horizontal panning of views. However, during vertical view rotation or zoom-in/zoom-out operations, the lookup table changes. In particular, the entries in the lookup table are required to be recalculated to find the new starting position and ending position on the image texture for each scanline. These recalculations result in a decrease of the playback speed.